G-XAI Bench

Here, we present G-XAI Bench, a data repository and open-source library of XAI-ready datasets that can be used to evaluate explainers for graph neural networks (GNNs). We have eleven total datasets: four synthetic datasets and seven real-world datasets. Many of these datasets feature ground-truth explanations which can be used for evaluating GNN explainers. Of note is the novel, flexible ShapeGraph dataset generator, which can generate graph datasets with varying properties and ground-truth explanations. We provide four instances of graphs generated from ShapeGraph in this repository, allowing for experiments varying homophily/heterophily, fair/unfair features, and large/small ground-truth explanation sizes. In addition, several chemistry datasets are included which have ground-truth explanations. Dataloaders are provided for each dataset in the open-source library G-XAI Bench, which can be found on our GitHub repository: https://github.com/mims-harvard/GXAI-Bench. Include in this repository are tutorials and reproduction code for extensive benchmarking of state-of-the-art GNN explainers.

Github repository link: https://github.com/mims-harvard/GXAI-Bench

Description of G-XAI Bench:

-----------------------------------------------------------------------------------------------
Filename				Description
-----------------------------------------------------------------------------------------------
alkane_carbonyl.npz	    Chemistry dataset for graph-level classification. Task is two-class
                            classification, 0 = has neither alkane or carbonyl functional 
                            groups, 1 = has alkane and carbonyl. Ground-truth explanations
                            are alkane and carbonyl functional groups. If there are multiple
                            combinations of alkane and carbonyl groups, then there will be
                            multiple ground-truth explanations, i.e., one for each combination.
                            Use G-XAI Bench functions to load the dataset (AlkaneCarbonyl 
                            class).
-----------------------------------------------------------------------------------------------
benzene.npz		        Chemistry dataset for graph-level classification. Task is two-class 
                            classification, 0 = does not have a benzene ring, 1 = has a benzene
                            ring. Ground-truth explanations are benzene rings. If there are 
                            multiple rings in one sample, there will be multiple ground-truth
                            explanations: one for each benzene ring present. Use G-XAI Bench 
                            functions to load the dataset (Benzene class).
-----------------------------------------------------------------------------------------------
Credit_Defaulter.tar.gz		Graph dataset with 30,000 nodes representing individuals connected 
                                based on the similarities of their spending and payment 
                                patterns. Age is a protected feature for each node. Task is 
                                node classification, 0 = no default on credit car, 1 = default 
                                on credit card. This dataset contains no ground-truth 
                                explanations and is used as a fairness and faithfulness 
                                benchmark dataset.
-----------------------------------------------------------------------------------------------
fluoride_carbonyl.npz	Chemistry dataset for graph-level classification. Task is two-class graph
                            classification, 0 = has neither fluoride nor carbonyl, 1 = has both
                            fluoride and carbonyl. Ground-truth explanations are fluoride and 
                            carbonyl functional groups. If there are multiple combinations of
                            fluoride and carbonyl groups in one sample, there will be multiple 
                            ground-truth explanations: one for each combination present. Use 
                            G-XAI Bench functions to load the dataset (FluorideCarbonyl class).
-----------------------------------------------------------------------------------------------
German_Credit.tar.gz	Graph dataset with 1,000 nodes representing clients in a German bank. 
                            Nodes are connected based on the similarities of their credit 
                            accounts. Gender is a protected feature for each node. Task is node 
                            classification, 0 = bad credit score, 1 = good credit score. This
                            dataset contains no ground-truth explanations and is used as a 
                            fairness and faithfulness benchmark dataset.
-----------------------------------------------------------------------------------------------
Mutagenicity.tar.gz 	Chemistry dataset for graph-level classification. Task is two-class graph
                            classification, 0 = not mutagenic, 1 = mutagenic. Ground-truth 
                            explanations are NH2, NO2, aliphatic halide, nitroso, and azo-type 
                            chemical groups, which are identified by Kazius et al. as 
                            toxicophores associated with mutagenicity. If there are multiple 
                            toxicophores within the same sample, there will be multiple ground-
                            truth explanations, each corresponding to a distinct combination of 
                            the toxicophores present. Use G-XAI Bench functions to load the 
                            dataset (Mutagenicity class).
-----------------------------------------------------------------------------------------------
Recidivism.tar.gz	    Graph dataset with 18,876 nodes representing defendants who were 
                            released on bail at U.S. state courts during 1990-2009. Nodes
                            are connected based on similarity of criminal records and 
                            demographics. Task is two-class node classification, 0 = did not 
                            receive bail, 1 = received bail. Race is considered as a protected 
                            attribute. This dataset contains no ground-truth explanations and 
                            is used as a fairness and faithfulness benchmark dataset.
-----------------------------------------------------------------------------------------------
SG-Base.pt	           Base ShapeGraph instance with homophilic node features, house-shaped
                            ground-truth explanations, and a fair protected feature. Task is 
                            two-class node classification, 0 = no motif in one-hop neighborhood, 
                            1 = motif in one-hop neighborhood. Use PyTorch to load Python class, 
                            i.e., `torch.load('path/SG-Base.pt')`.
-----------------------------------------------------------------------------------------------
SG-Heterophilic.pt	ShapeGraph instance with same structure as SG-Base, only with hetero-
                        philic node features. Task is two-class node classification, 0 = no 
                        motif in one-hop neighborhood, 1 = motif in one-hop neighborhood. Use 
                        PyTorch to load Python class, i.e.,
                        `torch.load('path/SG-Heterophilic.pt')`. Ground-truth explanations
                        are house motifs, as in SG-Base.
-----------------------------------------------------------------------------------------------
SG_SmallEx.pt		ShapeGraph instance with triangles as ground-truth explanations. Task is 
                            two-class node classification, 0 = no motif in one-hop neighborhood, 
                            1 = motif in one-hop neighborhood. Use PyTorch to load Python class, 
                            i.e., `torch.load('path/SG_SmallEx.pt')`. Ground-truth explanations 
                            are triangle motifs, i.e., a three-member cycle.
-----------------------------------------------------------------------------------------------
SG-Unfair.pt	    ShapeGraph instance with same structure as SG-Base, only with unfair
                        protected attribute. Task is two-class node classification, 0 = no motif 
                        in one-hop neighborhood, 1 = motif in one-hop neighborhood.Use PyTorch 
                        to load Python class, i.e., `torch.load('path/SG-Unfair.pt')`. Ground-
                        truth explanations are house motifs, as in SG-Base.
-----------------------------------------------------------------------------------------------